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16.  Abtfroct 


A validation  plan  is  presented  for  an  airside  simulation  model.  The 

plan  stresses  basic  principles  of  validation  and  inherent  problems 
associated  with  comparing  simulation  model  delay  estimates  with 
observable  real-world  data.  A methodology  is  proposed  that  consists 
of  three  major  steps:  (1)  evaluation  of  model  logic,  inputs,  and  outputs; 

(2)  comparison  of  model  estimates  with  collected  data;  and  (3)  a sensitivity 
analysis  of  the  delay  simulation  model.  A Model  Validation  Group,  established 
to  oversee  the  validation  process,  is  described.  Suggestions  are  given 
on  sources  of  data  on  airside  operations  for  model  inputs  and  for 
comparisons  with  model  estimates.  4^ 
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DELAY  MODEL  VALIDAFION  PLAN 
by 

William  J.  Dunlay,  Jr. 


I.  INTRODUCTION 


Purpose 

The  purpose  of  this  report  Is  to  present  a validation  plan  for  a 
fast-time,  stochastic,  delay  simulation  computer  model.  The  validation 
effort  is  part  of  Phase  I of  contract  No.  DOT-FA77WA-3961  with  Peat, 

Marwick,  Mitchell  & Co.  The  objective  of  the  validation  is  to  test 
whether  the  model  is  satisfactory  (to  the  Technical  Officer)  for  its 
intended  application  in  Phase  II  of  the  contract,  namely  for  delay 
estimation  at  seven  airports  in  support  of  six  Airport  Improvement  Task  Forces. 


Model  Validation  Group 

A Model  Validation  Group  has  been  appointed  by  Technical  Officer  to 
oversee  the  validation  process.  This  is  a very  significant  aspect  of  the 
validation  plan  since  a variety  of  expertise  is  required  to  evaluate  a model 
of  a system  as  complex  as  the  airport  airside  system. 

There  are  many  precedents  to  using  an  overseeing  committee  in  the 
validation  of  simulation  models.  In  fact.  Van  Horn  suggests  below  that  it 
is  part  of  an  ideal  validation: 

"Ideally  a comparison  test  should  handle  nonstationarity,  compensate 
for  noisy  data,  simultaneously  evaluate  a number  of  output  measures 
and  work  for  small  samples.  Does  such  a test  exist?  The  answer  is 
yes  if  one  is  willing  to  define  test  very  broadly.  The  test  is 
simple.  Find  people  who  are  directly  involved  with  the  actual 
process.  Ask  them  to  compare  actual  with  simulation  output."' 

The  Model  Validation  Group  consists  of  the  following  individuals: 

^Van  Horn,  R.  L.,  "Validation  of  Simulation  Results,"  Management  Science, 
Vol.  17,  No.  5,  Jan.  1971,  p.  252. 
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General  Considerations 

The  validation  of  any  computer  simulation  model  of  a complex  system 
is  a very  difficult  task.  It  is  a part  of  a more  general  problem,  namely 
the  validation  of  any  kind  of  model  or  hypothesis,  about  which  there  is  much 
literature  but  very  little  agreement.  One  textbook  on  computer  simulation 
states  that  "...the  problem  of  verifying  simulation  models  remains  today 
perhaps  the  most  elusive  of  all  the  unresolved  problems  associated  with 
computer  simulation  techniques."  Richard  L.  Van  Horn  mentions  two 
important  characteristics  of  the  validation  problem: 

"1.  The  objective  is  to  validate  a specific  set  of  insights  not 
necessarily  the  mechanism  that  generated  the  insights. 

2.  There  is  no  such  thing  as  'the'  appropriate  validation  procedure. 
Validation  is  problem  dependent."^ 

Van  Horn's  point  is  that  it  is  t^ie  major  attributes  of  the  particular 
processes  to  be  simulated  that  must  guide  the  general  approach  to  a validation. 

Validation  holds  a special  and  Important  role  in  computer  simulation 
models.  Unlike  most  analytical  models,  simulation  models  tend  to  conceal 
their  assumptions  and  internal  processes  from  the  casual  observer.  Further- 
more, the  nature  of  simulation  models  can  vary  dramatically.  For  example, 
as  Van  Horn  points  out,  "Tne  simple  statement  that  model  x is  a linear 
programming  model  conveys  a great  deal  of  information  about  its  structure, 
assumptions,  and  limitations.  The  statement  that  model  is  a simulation 

3 

conveys  virtually  no  information."  Therefore,  the  validation  of  a simula- 
tion model  requires  an  investigation  of  the  model's  internal  structure  in 

addition  to  comparing  the  input-output  transformations  generated  by  the 

2 

Naylor,  et  al..  Computer  Simulation  Techniques,  New  York:  John  Wiley 
& Sons,  Inc.,  1966,  p.  3l0. 

^Van  Horn,  R.  L. , p.  248. 
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model  to  those  generated  by  the  real  world. 

Any  model  validation  should  be  carried  out  In  two  basic  steps: 

(1)  a check  of  the  validity  of  the  assumptions  and  logic  of  the 
model ; and 

(2)  a comparison  of  the  estimates  produced  by  the  model  to  real 
world  observation. 

There  is  very  little  disagreement  among  modelers  that  both  of  these 
steps  are  required.  In  his  book  on  systems  analysis,  de  Neufvllle  states 
that  "...  statistical  analysis  cannot  be  a sufficient  test  of  any  model. 

The  validity  of  a systems  model  also  rests  on  the  plausibility  of  Its 

4 

a priori  theoretical  base." 

That  the  two  foregoing  steps  are  required  follows  from  the  fact  that 
it  is  the  predictive  power  of  the  mode!  that  is  of  concern,  not  the 
explanatory  power.  That  is  to  say,  it  is  not  sufficient  to  test  just  the 
goodness-of-fit  of  the  model  to  observed  data.  Naylor  states  that 
"...  the  ultimate  test  of  a computer  simulation  model  is  the  degree  of 
accuracy  with  which  the  model  predicts  the  behavior  of  the  actual  system 
(which  is  being  simulated)  in  the  future."^ 

Unfortunately , because  one  cannot  observe  the  future.  It  is  not 
possible  to  directly  validate  the  predictive  capabilities  of  a simulation 
model.  Instead,  one  must  rely  on  the  evidence  of  how  well  the  model  fits 
observable  data  coupled  with  the  evidence  of  how  well  the  logic  and 
assumptions  of  the  model  seem  to  make  it  extrapolatable  to  other  (non- 
observable) situations. 

^deNeufville,  R.,  Systems  Analysis  for  Engineers  and  Managers,  New  York: 
McGraw  Hill  Book  Co.,  1971,  p.  266. 

^Naylor,  et.  al . , p.  318. 


The  foregoing  difficulties  apply  to  situations  where  the  real-world 
situation  is  easy  to  ooserve  (measure).  “The  problem  of  model  validation 
becomes  even  more  difficult  if  the  available  data  about  the  ‘actual*  be- 
havior of  the  world  is  [sic]  itself  subject  to  error. This  certainly 
applies  to  the  simulation  model  being  considered.  Observed  values  of  delay, 
travel  time,  holds,  etc.,  are  subject  to  significant  field  measurement 
errors.  Even  if  these  quantities  could  be  accurately  measured,  they  are 
subject  to  large  random  fluctuations. 

There  are  a few  more  complications  that  apply  particularly  to  when  one 
tries  to  compare  delays  suffered  by  arrivals  in  the  airspace,  a very  impor- 
tant component  of  total  airside  delay,  as  estimated  by  the  model  to 
corresponding  real-world  values. 

First  of  all  it  is  difficult  to  separate  airspace  delays  due  to 
destination-airport  congestion  from  those  due  to  en  route  congestion  or 
ATC  instructions.  A second  and  closely  related  problem  is  that  those 
airspace  delays  that  are  attributable  to  the  destination  airport's  capacity 
constraints  are  not  all  incurred  at  one  point.  Such  delays,  for  example, 
may  be  incurred  en  route  at  the  advice  (say  speed  control  or  path  stretching) 
of  a controller  or  dispatcher,  i.e.,  delays  can  back  up  to  various  distances 
before  the  aircraft  arrives  at  the  terminal  airspace. 

Overview  of  Validation  Approach 

This  validation  must  proceed  in  spite  of,  but  also  cognizant  of,  the 
foregoing  inherent  problems  of  validation.  Towards  this  purpose,  the  valida- 
tion plan  incorporates  the  following  two  key  ideas: 


^Naylor,  et  al . , p.  318. 
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(1)  A variety  of  comparisons  should  be  made  between  model  estimates 
and  real-world  measures  rather  than  rely  on  just  a single  valida- 
tion variable.  By  doing  so  the  Model  Validation  Group  can  weigh 
all  the  evidence  In  deciding  whether  or  not  the  model  satisfac- 
torily fits  measured  data. 

(2)  It  should  be  recognized  that.  In  the  end,  the  decision  as  to  the 
model's  acceptability  for  Its  intended  application  Is  a subjective 
one  based  on  a combination  of  statistical  hypothesis  testing  and 
just  "eyeballing"  certain  aspects  of  the  model's  outputs,  logic, 
and  sensitivities.  Hypothesis  tests  should  be  conducted  In  such 

a way  that  they  don't  presume  to  make  the  decision  as  to  ac- 
ceptance or  rejection,  but  Instead,  simply  supply  a quantitative 
measure  of  goodness-of-f 1 t of  the  model's  estimates. 

Listed  below  are  the  three  major  steps  to  be  followed  in  the  valida- 
tion of  the  delay  simulation  model. 

(1)  Evaluation  of  the  model's  detailed  logic  and  assumptions,  the 
scope  and  kinds  of  Inputs,  and  scope  and  kinds  of  output. 

(2)  Evaluation  of  how  well  model  estimates  of  delays,  travel  times, 
and  flow  rates  compare  with  our  best  available  real-world  measure- 
ments of  these  variables. 

(3)  Evaluation  of  the  sensitivities  of  the  model  to  changes  In  certain 
key  Inputs  and  assumptions. 

Each  of  these  three  stages  of  the  approach  is  described  in  detail  in 


L 


the  following  sections. 
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II.  MODEL  LOGIC  AND  ASSUMPTIONS,  INPUTS,  AND  OUTPUTS 
Scope  and  Kinds  of  Inputs 

One  aspect  of  the  contractor's  simulation  model  that  should  be  evaluated 
is  the  required  inputs  for  its  application.  There  are  about  five  specific 
questions  about  the  inputs  that  should  be  addressed  as  listed  below: 

(1)  Are  the  inputs  sufficient  to  represent  the  operations  at  an 
airport?  In  answering  this  question  one  must  consider  the 
list  of  inputs  in  the  contractor's  User's  Manual  and  other 
inputs  provided  by  the  pre-processors. 

(2)  Are  the  inputs  sufficient  to  distinguish  among  the  different 
possible  runway-use  configurations  at  a major  airport  and 
also  the  different  airspace  routings  of  aircraft  to  these 
different  use  patterns? 

(3)  Are  the  inputs  sufficiently  sensitive  to  local,  i.e., 
airport-specific,  conditions? 

(4)  How  difficult  is  it  to  obtain  the  required  inputs?  Is  this 
excessive  given  the  expected  benefits  of  applying  the  model? 

(5)  How  sensitive  is  the  required  set  of  inputs  to  possible  future 
changes  in 

(a)  runway-use  configurations? 

(b)  aircraft  mix? 

(c)  terminal  building  size  and  configuration? 

(d)  noise  abatement  strategies? 

(e)  energy  and  fuel  conservation  measures  in  aircraft  operation? 

(f)  aircraft  separations  and  other  ATC  rules? 
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Model  Logic  and  Assumptions 

There  are  two  major  kinds  of  assumptions  made  in  any  simulation  model: 

(1)  simplifying  assumptions  and  (2)  statistical  assumptions. 

As  abstracts  of  the  real  world,  simulation  models  necessarily  in- 
volves simplifying  assumptions.  Examples  include  constant  aircraft  speeds 
on  approach,  runway  exit  used  independent  of  airline,  and  arriving-aircraft 
taxiing  route  dependent  only  on  runway  exit  and  destination  gate  (or  hold 
area).  These  assumptions  should  be  clearly  identified  and  listed  by  the 
contractor  to  facilitate  evaluation  by  the  Model  Validation  Group. 

There  are  three  major  types  of  statistical  assumptions.  The  first  is 
whether  a given  quantity  is  assumed  a fixed  constant  or  a random  variable. 

The  second  is  a probability  distribution  for  each  random  variable.  The 
third  type  of  statistical  assumption  has  to  do  with  the  statistical  depen- 
dencies among  the  various  random  variables.  For  example,  aircraft  approach 
speeds  are  assumed  a random  variable  with  a normal  distribution  with  para- 
meters dependent  upon  aircraft  class.  Furthermore,  in  assigning  approach 
speeds,  successive  aircraft  speeds  are  assumed  to  be  statistically  independent. 

All  such  assumptions  should  be  clearly  specified  in  the  contractor's 
presentation  of  the  model.  In  addition,  any  prior  empirical  validation  of 
the  assumptions  should  be  described.  The  validation  group  should  decide 
whether  any  additional  empirical  comparisons  are  desirable. 

The  model  logic  consists  of  the  foregoing  assumptions  and  the  relation- 
ships among  the  variables  of  the  airside  system  as  implied  by  the  way  the 
model  manipulates  the  variables.  These  relationships  and  manipulations 
should  be  evaluated  by  each  member  of  the  Model  Validation  Group  against 
his  knowledge  and  understanding  of  airfield  operations.  To  facilitate  such 
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an  evaluation,  the  contractor  should  provide  macrc-logic  flow  charts  that 
"walk  the  group  through"  the  simulation  showing  what  happens  to  a particular 
aircraft  from  the  time  it  enters  the  system  on  arrival  to  when  it  departs 
the  system.  The  contractor  should  describe  the  types  of  aircraft  interactions 
and  other  airside  situations  that  the  model  can  handle.  The  Model  Validation 
Group  should  judge  whether  these  interactions  and  situations  are  representa- 
tive of  conditions  actually  encountered  at  large  complex  airports. 

At  the  meeting  where  the  contractor  discloses  the  model  logic  to  the 
Model  Validation  Group,  the  contractor  should  also  describe  the  details 
of  prior  validations  of  the  model.  All  prior  sensitivity  analyses  of  the 
model  should  also  be  presented  by  the  contractor  at  that  meeting.  Further- 
more, the  contractor  should  demonstrate  that  the  model  is  operational  on  a 
time  sharing  computer  system  at  the  time  of  the  model  disclosure  meeting. 

The  overall  behavior  of  the  airside  system  being  simulated  is  strongly 

influenced  by  air  traffic  controllers  and  dispatchers  acting  as  decision 
makers  and  information  processors.  The  fact  that  these  influences  are 
not  explicitly  modeled  in  the  contractor's  fast-time  model  complicates 
the  process  of  validation.  There  are,  however,  implicit  elements  of  the 
model  logic  designed  to  reflect  certain  types  of  controller  and  dispatcher 
actions.  These  elements  will  be  evaluated  as  to  their  realism  by  the 
Model  Validation  Group. 

Scope  and  Kinds  of  Output 

The  quality  of  the  outputs  can,  of  course,  be  no  better  than  that  of 


the  inputs  and  logic.  Nevertheless,  one  evaluation  criteria  for  the  model 
is  level  of  output  detail  that  it  presents  about  the  1 evel -of-service 


10 


experienced  on  an  airfield  by  a given  demand  pattern.  For  example,  does 
the  model  provide  delay  information  by  cause,  location,  type  of  aircraft, 
airline,  etc.?  Is  there  sufficient  flexibility  in  cross  tabulating  and 
aggregating  the  outputs  for  subsequent  analysis?  To  answer  these  questions 
the  Model  Validation  Group  should  evaluate  the  raw  model  outputs  and  the 
outputs  of  the  post-processors. 

Besides  outputs  related  to  delays,  the  validation  group  should  also 
investigate  the  options  for  obtaining  output  data  on  level -of-service 
measures  such  as  queue  lengths,  travel  times,  flow  rates,  and  known 
bottlenecks. 

Approach  to  Validating  Inputs,  Logic,  and  Outputs 

It  is  proposed  that  the  validation  of  the  model  inputs,  logic,  and 
outputs  be  accomplished  by  a contractor  presentation  to  a working  sub-group 
of  about  eight  persons. 

The  contractor  should  provide  macro-logic  flow  charts  of  the  model 
to  the  members  of  the  working  sub-group  ahead  of  time  so  that  they  can 
examine  the  logic  of  the  model  in  detail.  Members  should  then  submit 
written  questions  to  the  contractor  at  least  one  week  in  advance  of  the 
presentation  so  that  the  contractor  can  focus  and  structure  his  presentation 
on  the  issues  raised  by  the  questions.  In  addition,  the  presentation  should 
include  a description  of  all  prior  validations  of  the  model  logic  and 
assumptions. 

The  contractual  requirement  for  the  model  is  that  it  be  a fast-time, 
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stochastic  simulation  which  simulates  arriving  aircraft  within  terminal 
airspace,  landing  and  movement  through  the  taxiway  system  up  to  and  including 
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' operations  in  the  gate  area  and  similarly  through  the  departure  phases. 

Operation  of  the  delay  model  should  provide  information  which  includes 
the  magnitude  and  time  distribution  of  airfield  delay. 

As  part  of  the  full  and  complete  disclosure  of  the  workings  and  code 
of  the  model, the  Chicago  Model  Validation  Group  will  require  a description 
of  the  airport  operation  characteristics  that  are  simulated  in  the 
model,  e.g.,  runway  occupancy,  approach  aircraft  interaction  with  other 
aircraft,  gate  management  and  operations  for  different  gate  configurations, 
taxiway  usage  (intersections,  aircraft  taxi  speeds,  etc.),  departure 
interaction  and  interarrival  gap  spacing  based  on  departure  queues.  This 
information  will  permit  comparison  with  the  coded  logic  to  verify  that  the 
described  operations  are  simulated  by  the  logic.  Secondly,  tne  group  would 
determine  during  this  effort  if  the  simulation  can  represent  real-world 
operations  at  Chicago. 

It  is  often  desirable  to  check  one  of  the  very  basic  elements  of  a 
simulation  model's  logic,  namely  its  arithmetic,  by  relaxing  all  of  the 
stochastic  assumptions  of  the  model  and  using  it  to  solve  a very  simple 
(even  trivial),  hypothetical  example  that  can  be  checked  by  hand  or  by 
using  simple  deterministic  models.  The  contractor  should  describe  in  his 
presentation  any  such  checks  of  the  model  that  were  performed  in  the  prior 
construction  and  development  of  the  model.  The  validation  group  should  then 
decide  whether  or  not  any  further  checking  of  this  type  would  be  desirable. 
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III.  EMPIRICAL  VALIDATION  OF  MODEL  OUTPUTS 

As  stated  earlier,  a variety  of  comparisons  will  be  made  between  model 
estimates  and  actual  observed  data;  in  particular,  comparisons  based  on  the 
following  variables; 

(1)  airspace  delays  to  arrivals  and  departure  delays 

(2)  ground  travel  times  for  both  arrivals  and  departures, 

(3)  aircraft  flow  rates 

(4)  departure  queues  , and 

(5)  penalty  box  (holding)  and  pushback  delays. 

The  first  three  variables  require  some  additional  explanation;  this  is 
presented  below. 

Airspace  Delays  to  Arrivals  and  Departure  Delays 

If  data  on  airspace  delays  are  used  as  a validation  variable  it  is 
essential  to  know  the  specific  runway  configurations  in  use  when  the  delays 
were  incurred.  This  information  can  be  obtained  either  from  tower  records 
or  from  direct  observations  made  in  the  tower.  The  problems  associated 
with  obtaining  an  adequate  sample  of  delay  data  for  particular  runway-use 
configurations  will  severely  limit  the  number  of  runway-use  configurations  tnat 
can  be  validated  especially  at  an  airport  like  O'Hare  where  conditions  are  so 
variable. 

The  problem  of  obtaining  an  adequate  sample  for  a given  runway  configura- 
tion is  illustrated  in  Fig.  1. 
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Fig.  1 Occurrence  of  Particular  Runway  Use  Configurations, 
A,  B and  C. 


The  shaded  bars  in  Fig.  1 represent  the  time  during  which  three  runway 
configurations  (A,  B,  and  C)  are  in  use.  Suppose  it  is  desired  to  observe 
a given  configuration,  say  A,  during  approximately  the  same  time  period 
for  a sample  of  n days. 


Note  that,  for  all  four  days  of  Fig.  1,  there  is  a common  2-hour 
period  (see  dashed  lines)  during  which  configuration  A is  used.  Configuration 
B,  is  not  so  repetitive;  it  is,  however,  used  during  a common  2-1/2  hour 
period  on  Days  2 and  4;  similarly  for  configuration  C. 

It  is  important  to  note  that  the  delays  encountered  in  a given  time 
period  (especially  at  the  beginning  of  the  period)  with  a given  runway  configura- 
tion may  depend  heavily  on  the  runway  configurations  used  the  immediately 
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preceding  tine  period.  For  example,  there  may  be  aircraft  left  over  from 
a lower-capacity  runway  configuration  in  the  preceding  period;  thus 
delays  in  the  beginning  of  the  period  would  be  higher  than  expected  and 
would  gradually  decline.  On  the  other  hand,  if  the  runway  configuration  of  the 
preceding  period  was  less  congested,  it  will  take  a while  for  the  delays  to 
build  up  to  be  representative  of  the  configuration  now  being  considered. 

In  Figure  1,  note  that  configuration  A is  preceded  by  configuration  B 
on  Day  1_,  and  by  configuration  C on  the  other  three  days.  Note  further  that, 
for  the  common  two-hour  period  in  which  A is  used  on  all  four  days;  on  Days 
2 and  4 the  two-hour  period  is  when  A is  just  beginning  to  be  used;  on  Days 
1 and  3 the  two-hour  period  falls  near  the  end  of  the  interval  during  which 
configuration  A is  being  used.  These  factors  are  additional  complications 
to  the  process  of  choosing  appropriate  samples  of  actual  data  for  comparison 
with  model  estimates. 

For  the  reasons  cited  above,  it  is  probably  not  feasible  to  obtain  ade- 
quate samples  of  identical  time  periods  on  successive  (assumed-independent) 
days.  Instead,  samples  will  consist  of  a number  of  observations  made  on  in- 
dividual days,  i.e. , in  successive  time  intervals  on  the  same  day.  Thus, 
different  days  will  represent  different  samples  to  be  treated  separately 
rather  than  averaged  together. 

The  foregoing  treatment  may  be  characterized  as  time  series  analysis. 

More  will  be  said  about  the  particular  time  series  analysis  assumptions  and 
techniques  recommended  for  this  validation  later  in  this  report. 

Ground  Travel  Times 

Travel  times  accumulated  by  the  model  are  fixed  at  a minimum  level  by 
the  model  input  for  the  airfield  layout.  They  will  increase  depending  upon 
runway  occupancy  time  and  delays.  Average  travel  times  can  be  calculated 
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without  model  operation  directly  from  the  model  input  by  summing  all  paths 
to  the  gates  from  the  runways  and  assigning  weights  based  on  the  exit  taxiway 
utilization  factors.  Average  travel  times  are  determined  by  and  reflect 
the  conditions  of  the  airport  and  runway-use  configurations.  The  results 
of  the  model  output  for  negligible  delays  can  serve  as  calibration  data. 

The  addition  of  delays  to  travel  times  in  the  model  logic  appears  to  be  an 
evaluation  factor  for  consideration  by  the  Model  Validation  Group. 

Travel  times  (both  in  the  air  and  on  the  ground)  are  an  important  output 
of  the  simulation  model  because  they  may  differ  for  different  runway  confi- 
gurations. Thus,  travel  times  are  an  important  level-of-service  measure  of 
a particular  runway-use  configuration  that  should  be  considered  along  with 
delays  when  comparing  configurations.  It  is  important,  therefore,  that  the 
model  be  able  to  predict  differences  in  the  travel  times,  both  in  the  ap- 
proach airspace  and  on  the  ground,  associated  with  alternative  configurations. 

Airlines  have  taxi-in  times  measured  from  wheels-on  to  the  gate  and  taxi- 
out  times  from  the  gate  to  wheels-off.  These  differ  slightly  from  the  model 
outputs;  the  discrepancies,  however,  are  small  (say  5-10  seconds)  and,  be- 
cause they  are  relatively  constant,  could  be  factored  out  of  airline  data. 

The  travel  times  produced  by  the  model  are  random  variables.  For  com- 
parison with  the  model  estimates,  a random  sample  of  airline  data  on  ground 
travel  times  will  have  to  be  obtained.  The  airline  data  should  be  field 
checked  for  accuracy  and  to  see  if  the  actual  taxiway  routings  and  the  model 
routings  are  comparable.  If  it  is  judged  that  the  airline  data  are  not 
satisfactory,  then  field  measurements  of  travel  times  will  have  to  be  used. 
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Aircraft  Flow  Rates 

Aircraft  flow  rates  are  recommended  as  a third  validation  variable. 

Flow  rates  accumulated  by  the  model  are  a function  of  the  number  and  type  of 
aircraft  arriving  in  a given  sequence  at  the  runway  and  the  separation 
standards  employed  in  the  input.  The  flow  rates  are  limited  by  the 
capacity  of  the  runway-use  configuration. 

The  number  of  operations  accomplished  in  various  size  time  intervals 
(e.g.,  a particular  fifteen-minute  or  a one-hour  period,  the  morning  peak, 
the  whole  day)  as  estimated  by  the  model  should  be  compared  to  corresponding 
field  measurements  of  these  operation  rates.  This  should  be  done  for  both 
arrivals  and  departures.  The  contractor  should  also  present  details  of  prior 
validations  of  the  model's  ability  to  accurately  estimate  flow  rates. 

Statistical  Treatments 

One  of  tne  most  difficult  and  troublesome  aspects  of  validation  is  the 
statistical  comparison  of  model  estimates  with  observed,  "real-world"  data. 

This  can  take  many  forms.  According  to  Van  Horn,  "Often  simple  comparisons 
of  means,  ranges  and  variances  and  graphical  comparison  of  distributions  or 
time  behavior  will  capture  most  of  the  available  information."^  Going 
beyond  this  and  doing  statistical  hypotheses  testing  is  possible,  but  great 
care  must  be  taken  in  selecting  appropriate  tests  for  this  purpose. 

The  literature  contains  frequent  warnings  about  the  statistical  nature  of 
the  output  of  simulation  models.  Hsu  and  Hunter,  for  example,  warn  that,"... 
data  from  many  simulation  models  are  often  not  serially  independent  of  time, 

O 

a fact  which  seriously  affects  the  validity  of  the  (standard  statistical)  tests." 

^Van  Horn,  p.  252. 

8hsu,  D.  A.  and  Hunter,  J.  S.,  "Analysis  of  Simulation-Generated  Responses 
Using  Autoregressive  Models,"  paper  accepted  for  Management  Science,  1977,  p.  2. 
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Fishman  and  Kiviat  similarly  point  out  that,  "As  simulation  data  are  generally 
autocorrelated,  an  investigator  cannot  apply  the  statistical  tools  commonly 

9 

used  for  studying  independent  observations." 

The  autocorrelation^^  problem  mentioned  in  the  foregoing  caveats  cannot 
be  ignored.  Fishman  and  Kiviat  go  on  to  say  that,  "Ignoring  autocorrelation 
is  clearly  unacceptable,  since  the  reliability  of  the  sample  means  and 

variances  are  thereby  overestimated."^^  Besides,  as  Hsu  and  Hunter  point  out, 

"...serial  correlation  in  time  is  itself  an  important  characteristic  of  the 

system  being  simulated,"  that  can  be  compared  statistically  to  the 

corresponding  serial  correlation  structure  of  the  real  world  data  as  part 

12 

of  the  validation  of  model  outputs. 

Based  on  the  above  discussions,  there  are  two  principal  candidate  methods 
for  use  in  a statistical  analysis  of  the  output  of  the  contractor's  fast-time 
simulation  model  vis-a-vis  observed  data,  one  proposed  by  Hsu  and  Hunter  and 
the  other  proposed  by  Fishman  and  Kiviat.  Both  of  these  methods  are  time- 
series  methods  that  consider  the  autocorrelation  structure  of  the  simulated 
data  and  the  observed  data. 

The  method  of  Hsu  and  Hunter  is  an  autoregressive  time  series  model  that 

simultaneously  compares  means,  variances,  and  autocorrelation  structures  of  two 

time  series.  More  precisely,  "...an  inferential  statistic. ..  is  used  to  compare 

two  time  series  simultaneously  with  respect  to  their  estimated  autoregressive 

parameters  and  variances.  A second  inferential  statistic. .. is  then  employed 

to  examine  the  differences  in  the  means  of  the  two  autoregressive  time  series." 

5 

Fishman,  G.  S.  and  Kiviat,  P.  J.,  "The  Analysis  of  Simulation-Generated 
Time  Series,"  Management  Science,  Vol . 13,  No.  7,  March,  1967,  p.  526. 

^*^Autocorrelation  is  a measure  of  the  linear  dependence  of  a process  on  its  past. 

^^Fishman  and  Kiviat,  p.  526. 

^^Hsu  and  Hunter,  p.  2, 

1 '3 

Hsu  and  Hunter,  p.  3 . 
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Appendix  D shows  an  example  of  applying  the  Hsu-Hunter  method  to  an  air 
traffic  control  model. 

Fishman  and  Kiviat  suggest  applying  "spectral  analysis"  to  the  study 
of  time  series  data  generated  by  simulation  models.  They  use  an  autocorre- 
lation function,  a spectral  density  function,  and  a statistic  called  "cor- 
relation time"  as  a statistical  description  of  the  two  time  series  being 
compared.  The  variance  of  the  sample  mean  of  each  time  series  is  shown 
by  Fishman  and  Kiviat  to  depend  on  these  statistics  and,  once  obtained, 
can  be  used  as  the  basis  for  hypothesis  testing  that  involves  comparing 
the  spectral  density  functions  of  the  simulated  and  observed  time  series. 

One  of  the  two  foregoing  methods  will  be  used  to  compare  the  various 

time  series  output  by  the  contractor's  model,  e.g.,  delays,  travel  times, 

queue  lengths,  flow  rates,  to  the  corresponding  measured  time  series  for 

these  quantities.  Both  methods  involve  the  assumption  of  a covariance-sta- 
14 

tionary  process.  Comparisons  will  be  made  of  single  realizations,  i.e., 
the  model  output  for  a single  random  number  seed,  versus  data  observed  on 
an  individual  day,  and  also  of  averages  over  several  random  number  seeds 
and  days.  The  sum  total  of  these  statistical  comparisons  will  enable  the 
Model  Validation  Group  to  make  a reasonable  judgment  as  to  how  closely  the 
simulation  output  approximates  the  real-world  data  collected  at  O'Hare. 

It  is  assumed  in  the  foregoing  statistical  analysis  that  the  random 
number  streams  corresponding  to  the  different  "seeds"  are  not  correlated. 
The  contractor  should  guard  against  choosing  seeds  that  give  streams  of 
random  numbers  displaced  by  only  a small  number  of  values.  In  his  presen- 
tation of  the  model  logic,  inputs  and  outputs  to  the  eight-man  working 

14 

A convariance-stationary  process  is  one  in  which  neither  the  co- 
variance  structure  nor  the  expected  value  of  the  time  series  is  a function 
of  time. 
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sub-group,  the  contractor  should  describe  how  the  different  random  num- 
ber seeds  are  chosen  and  whether  or  not  a check  has  been  made  that  the 
different  streams  are  uncorrelated. 

Concluding  Remarks  - Empirical  Verification 

By  comparing  model  estimates  of  airspace  delays,  ground  travel  times, 
and  flow  rates  with  measured  data,  one  can  base  an  evaluation  of  the  good- 
ness-of-fit  of  the  model  on  a variety  of  empirical  evidence.  The  decision 
as  to  the  adequacy  of  the  model  for  Phase  II,  however,  must  also  be  based 
on  an  evaluation  of  the  model's  logic  and  its  fine-grained  sensitivity  as 
described  in  the  next  section. 


A 


( 
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IV.  SENSITIVITY  ANALYSIS  OF  THE  MODEL 

This  third  phase  of  the  model  validation  is  aimed  at  exploring  certain 
properties  of  the  model  itself.  It  will  probably  not  involve  any  field 
data  collection  or  statistical  hypothesis  testing. 

A sensitivity  analysis  usually  involves  evaluating  the  change  in  one 
or  more  key  outputs  (e.g.,  estimated  delay)  resulting  from  systematic 
changes  in  one  or  more  input  parameters.  There  are  several  reasons 
why  one  might  want  to  do  this.  One  is  that  if  the  model  outputs  are  very 
sensitive  to  small  changes  in  one  of  the  input  parameters,  then  that 
parameter  will  have  to  be  measured  very  accurately  and  assumptions 
about  that  parameter  closely  scrutinized;  if  not,  less  measurement 
accuracy  is  satisfactory. 

A second  reason  is  to  evaluate  how  extrapolatable  the  model  is  to  new, 
non-observable  situations  by  systematically  varying  one  or  more  inputs 
and  then  judging  the  resulting  output  changes  predicted  by  the  model 
against  what  we  would  expect  to  happen  from  our  knowledge  and  experience. 

Thus,  the  sensitivity  of  the  model  is  a very  important  aspect  of  its 
logic,  For  example,  one  may  wish  to  examine  the  delay  (flow  rates, 
gate  congestion,  etc.)  that  result  from  incrementally  adding  new  aircraft 
to  the  existing  demand.  Or  one  may  want  to  determine  whether  the  model 
can  reasonably  predict  the  effect  of  a major  perturbation  such  as  a 

sudden  drop  in  ceiling  and  visibility  that  is  known,  a priori,  based  on 
past  experience  at  a particular  airport  to  have  a dramatic  effect  on  delays. 

A third  possible  reason  for  a sensitivity  analysis  is  to  evaluate 
how  sensitive  the  results  are  to  simplifying  and  statistical  assumptions. 

Suppose  an  assumption  is  made  that  is,  for  one  reason  or  another,  not 
well  documented.  If  it  turns  out  that  the  model  output  is  very  sensitive 
to  small  deviations  from  our  assumption,  then  an  effort  should  be  made 
to  further  check  (and  possibly  revise)  the  assumption.  This  is  another 
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indication  that  the  sensitivity  analysis  is  a very  important  adjunct 
to  our  evaluation  of  the  model  logic. 

The  eight-man  working  sub-group,  described  earlier  under  model  logic, 
should  also  oversee  the  fine-grain  sensitivity  analysis.  They  should  decide 
which  parameters  to  fix  and  which  to  vary  for  the  sensitivity  demonstration. 
Furthermore,  the  contractor  should  describe  in  detail  all  sensitivity 
analyses  done  during  prior  model  development  and  demonstrations  during  the 
disclosure  of  model  inputs,  outputs,  and  logic  to  the  sub-group. 
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V.  DATA  COLLECTION 

In  this  validation  exercise  data  will  be  required  for  two  principal 
purposes: 

(1)  to  provide  the  necessary  inputs  for  the  model,  and 

(2)  to  provide  a sample  of  observed  data  against  which  model 
estimates  can  be  compared. 

The  model  input  data  can  be  further  subdivided  into  four  categories: 

(1)  model  specification  data,  (2)  airside  specification  data,  (3) 
demand  specification  data,  and  (4)  airport  operation  data.  Detailed  lists 
of  each  of  these  four  categories  are  given  in  Tables  1 through  4.  These 
tables  also  suggest,  for  each  data  item,  primary  and  secondary  data 
sources  and  the  party  responsible  for  obtaining  the  data.  Table  5 presents 
a similar  description  of  data  required  for  comparison  with  model  outputs. 

A subsequent  plan  for  data  collection,  reduction,  and  analysis  will 
present  greater  detail  on  the  actual  methods  and  equipment  to  be  employed, 
manpower  required,  runway  configurations  to  be  studied,  etc.  It  is  likely 
that  some  minor  adjustments  will  be  made  to  the  descriptions  of  Tables  1 
through  5 as  the  data  collection  progresses. 

Following  the  approximate  three  week  period  of  data  collection,  the  data 
reduction  can  proceed  in  two  phases: 

(1)  reduction  of  data  for  model  inputs 

(2)  reduction  of  comparison  data  on  model  outputs. 

Ihib  reduction  effort  will  be  very  time  consuming  if  it  is  not  carefully 
planned  in  advance.  Existing  computer  programs  for  reading  and  manipulating 
the  data  (say  from  the  contractor  or  other  sources)  should  be  used  to  the 
fullest  extent  possible.  Detailed  data  collection  procedures  should  be  planned 
with  the  subsequent  reduction  requirements  uppermost  in  mind. 


Processing  Options  Options  to  either  print  Specified  by  MVG  — MVG 

input  only  or  to  compute 
a variable  number  of 
data  sets 


TABLE  2.  INPUT  DATA  - AIRSIDE  SPECIFICATION 
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An  important  aspect  to  the  data  collection  to  consider  is  the  definition 
of  terms  of  the  delay  measurements  as  reflected  in  the  model.  The  measure- 
ments at  the  facility  must  comply  with  the  output  of  the  model;  event  times 
specified  in  the  model  must  be  recorded  at  the  facility.  (For  example, 
time  of  arrival  and  departure  time  from  gate  must  be  defined,  collected, 
and  reduced  before  comparison  with  the  model  output.) 

The  form  of  the  data  extracted  will  be  dependent  upon  the  definition  of 
the  delay  factors  and  parameters  expressed  in  the  simulation  model.  The 
following  table  (Table  6)  lists  a set  of  event  times  along  with  a description 
of  the  position  of  the  aircraft,  what  the  aircraft  is  doing,  and  what  the 
aircraft  is  about  to  do  in  the  simulation.  This  description  of  the  event 
times  in  the  simulation  permits  the  measured  data  to  be  sectioned  into  the 
various  categories  of  delay  at  the  airport.  These  event  times  of  the 
actual  simulation  model  must  be  clearly  defined  and  reconfirmed  (after 
disclosure  of  code  and  listings  from  the  contractor)  before  data  reduction 
to  insure  that  delay  accumulations  may  be  compared  with  the  model  output  at 
the  conclusion  of  the  validation  procedure. 

The  concept  of  matching  field  measurements  at  the  airport  with  the  model's 
definitions  of  airport  operations  applies  particularly  to  the  travel  times. 

The  model  simulation  calculates  travel  time  by  sumning  the  total  times  an 
aircraft  occupies  individual  links  on  the  route  to  the  gate,  holding  area  or 
runway  takeoff  point.  Therefore,  aircraft  travel  times  should  be  measured 
at  the  airport  as  shown  in  Table  7. 


TABLE  6.  DESCRIPTION  OF  SIMULATION  MODEL  EVENTS 
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Aircraft  attempting  to  Aircraft  in  gate  (scheduled  Next  Event:  Pushback  from  gate 

depart  gate  after  or  general  aviation)  Operation:  Link  by  link  movement 

layover  from  previous  to  takeoff  link 


TABLE  7.  AIRPORT  TRAVEL  TIME  MEASUREMENTS 
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Description  of  Measurement  Points 

Aircraft  arrives  at  runway  threshold  and  stops  at  gate 

Aircraft  arrives  at  runway  threshold  and  stops  at 
holding  area 


Aircraft  starts  movement  from  holding  area, 
and  stops  at  gate 

Aircraft  begins  pushback  (receives  controller 
clearance)  and  stops  just  before  takeoff  roll 
(not  always  at  same  point  on  airport). 

The  accumulation  of  hourly  delay  totals  will  involve  the  extraction  and 
combination  of  data.  Delay  measurements  will  be  classified  into  areas  which 
agree  with  the  model  output.  There  is  a need  to  define  the  measurement 
points  for  each  classification  of  delay  used  in  the  model.  In  addition, 
certain  conditions  which  occur  during  simulation  (such  as  queues)  determine 
the  classification  of  the  delay.  Table  8 lists  the  model  delays  and  the 
corresponding  events  which  define  them  or  the  conditions  which  classify  them. 


Measurement 

Travel  time  from 
Runway  to  Gate 

Travel  time  from 
Runway  to  Holding 
Area 

Travel  time  from 
Holding  Area  to  Gate 

Travel  time  from 
Gate  to  Runway 


TABLE  8.  MODEL  DELAY  CLASSIFICATIONS 


Item 

a 

b 


c 


d 


e 

f 

9 
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Delay 

Arrival  Runway  Delay 

Arrival  Gate  Delay 
(Holding  area  delay) 


Departure  Gate  Delay 
(accumulated  under  taxi- 
out  delay) 


Taxi -in  Runway  Crossing 


Taxi -Out  Runway  Crossing 
Arrival  Taxi-In  Delay 


Departure  Taxi -Out  Delay 


Departure  Runway  Delay 


Event  or  Simulation  Condition 


Aircraft  attempts  to  land  at  scheduled  Time 
of  Arrival  (event) 

Aircraft  at  exit  taxiway  (event)  is  denied 
a gate  and  routed  to  a holding  area.  Delay 
is  terminated  at  the  time  (event)  of  the 
next  available  gate 

Aircraft  attempts  pushback  but  is  delayed 
because  of  presence  in  gate  area  of  an  in- 
bound aircraft  (attempt  is  repeated  when  de 
lay  value  is  exceeded  in  the  sequence  of 
events  in  the  simulation) 

Arriving  aircraft  is  in  taxiway  crossing 
link  awaiting  clearance  across  runway 
(aircraft  in  queue  behind  this  link 
accumulate  taxi -in  delays) 

Same  as  (d)  above  for  departing  aircraft 

Aircraft  moves  from  link  to  link  and  if 
delayed  accumulates  taxi  delay 

Aircraft  moves  from  link  to  link  and  if 
delayed  acculates  taxi-out  delay  (departure 
gate  delays  are  also  accumulated  under 
taxi-out  delays) 

Aircraft  is  in  departure  link  or  in 
departure  queue  awaiting  departure 
clearance 
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APPENDIX  A 


Summary  of  Steps 

I.  Plan  for  Contractor  Disclosure  and  Recommendation* 

A.  Model  Inputs 

B.  Model  Outputs 

C.  Model  Logic 

D.  Prior  Validation  of 

1.  Model  Logic 

2.  Model  Sensitivity 

3.  Model  Outputs 

4.  Model  Assumptions 

II.  Selection  of  Validation  Variables 

A.  Arrival  Airspace  Delays 

B.  Departure  Delays 

C.  Ground  Travel  Times 

D.  Aircraft  Flow  Rates 

III.  Specification  of  Sensitivity  Analysis* 

A.  Variables  to  be  held  fixed 

B.  Variables  to  be  systematically  varied 

C.  Response  Variables  to  be  investigated 


To  be  accomplished  during  contractor's  presentation.  Beforehand,  the 
contractor  submits  macro-logic  flow  charts  and  the  working  Sub-group  submits 
questions  on  inputs,  logic  and  outputs.  Working  Group  specifies  sensitivity 
parameters  at  end  of  presentation. 
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APPENDIX  B 


Meeting  Agendas 

I.  Presentation  and  Adoption  of  Validation  Plan 

A.  Date  - May  18,  1977 

B.  Place  - Washington,  D.  C. 

C.  Attendance  - Full  Validation  Group 

D.  Agenda 

1.  Presentation  and  Discussion  of  Strawman  Plan  - W.  J.  Dunlay 

2.  Comments,  Suggestions,  Revisions  - Validation  Group 

3.  Adoption  of  Final  Plan  - Validation  Group 

4.  Discussion  of  Data  Collection  and  Reduction 

a.  collection  assignments 

b.  reduction  assignments 

c.  coordination 

d.  schedule  and  detailed  plan 

5.  Discussion  of  Next  Meetings 

a.  Full  Validation  Group 

b.  Working  Sub-Group 

II.  Contractor  Disclosure  and  Presentation 

A.  Date:  within  two  weeks  after  adoption  of  validation  plan 

B.  Place:  San  Mateo,  Calif. 

C.  Attendance:  Working  Sub-Group  of  approximately  eight  people 

and  a contractor  representative. 

D.  Preparation: 

1.  Contractor  should  provide  members  of  working  group  with 


copies  of  macro-logic  flow  charts. 
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Preparation  (continued) 

2.  Members  should  study  the  flow  charts  and  submit  written 
questions  to  the  contractor. 

3.  Contractor  should  design  presentation  to  be  responsive 
to  submitted  questions. 

E.  Agenda: 

1.  Overview  of  Details  of  the  Model  Logic,  Inputs,  and  Out- 
puts - Contractor 

2.  Review  and  Discussion  of  Submitted  Questions  - Contractor 
anJ  Group 

3.  Further  Questions  and  Answers  on  Model  Logic,  Inputs  and 
Outputs  - GrouD  and  Contractor 

4.  Review  of  prior  Model  Validations  (empirical  verifications)  of 

a.  Model  Assumptions 

b.  Model  logic-arithmetic 

c.  Model  Outputs  - Contractor 

5.  Discussion  and  Questions  on  prior  verifications  - Group  and 
Contractor 

6.  Review  prior  sensitivity  analyses  of  model  - Contractor 

7.  Specification  ana  design  of  Sensitivity  Demonstration  - Group 

a.  Selection  of  variables  to  be  held  fixed 

b.  Selection  of  variables  to  be  systematically  varied 

c.  Selection  of  response  variables  to  be  evaluated. 

d.  Evaluation  Procedure 


I 

J 
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Agenda  (continued) 

8.  Discussion  of  Data  Collection 
Requirements  for 

a.  Model  Inputs 

b.  Model  Validation  - for  comparison  with  model  outputs 

III.  Detailed  Data  Collection  and  Reduction  Plan 

A.  Data:  within  two  weeks  after  adoption  of  Model  Validation  Plan 

B.  Place:  Washington,  D.C. 

C.  Attendance:  Sub-Group 

D.  Preparation:  Members  should  study  validation  plan  as  adopted 
vis-a-vis  the  resources  of  the  group  for  data  collection  and 
reduction 

E.  Agenda: 

1.  Review  of  data  needs  as  implied  by  model  validation  plan 

2.  Discussion  of  manpower  needs  and  a schedule  of  collection 
and  reduction  activities 

3.  Identification  of  equipment,  software,  and  manpower  resources 
of  the  Group. 

4.  Assignment  of  specific  tasks  to  Group  members 

5.  Discussion  of  data  collection  procedures,  forms,  and  sample 
sizes  - Group  and  Contractor 

6.  Discussion  of  data  reduction  methods  and  desired  format  of 
reduced  data  - Group  and  Contractor 
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Agenda  (continued) 

7.  Specification  of  coordination  procedures  to  be  followed 
during  data  collection  and  reduction 

8.  Outline  of  data  collection  and  reduction  plan  and  schedule 

j 

I 


L 


A 
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APPENDIX  C 

STEP-BY-STEP  SUMMARY 


Evaluation  of  Model  Logic.  Assumptions,  Inputs,  and  Outputs 

A.  Begins: 

As  soon  as  preparations  (see  below)  can  be  made,  but  no  later  than 
two  weeks  after  adoption  of  validation  plan. 

B.  Methodology: 

The  evaluation  will  be  made  by  a working  subgroup  of  about  8 per- 
sons. No  formal  evaluation  criteria  will  be  used.  The  evaluation 
will  be  based  on  the  knowledge  and  experience  of  the  members  of  the 
working  subgroup. 

C.  Preparations: 

(1)  Contractor  should  provide  members  in  advance  with  whatever 
written  documentation  of  the  model  is  available,  including  macro- 
logic flow  charts. 

(2)  Members  should  familiarize  themselves  with  the  written 
documentation  and  submit  written  questions  if  they  have  any. 

(3)  Contractor  should  present  to  the  working  subgroup  details 
of  the  model  logic,  assumptions,  inputs  and  outputs  and  copies  of 
macro-logic  flow  charts. 
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D.  Simultaneous  Activities: 

The  preparation  of  the  detailed  data  collection  plan  can  begin 
immediately  after  validation  plan  adoption  and  may  be  carried  on 
simultaneously  with  the  evaluation  of  model  logic,  assumptions, 
inputs  and  outputs. 

E.  Subsequent  Activities:  i 

The  specification  and  evaluation  of  the  sensitivity  analysis  j 

by  the  8-person  working  subgroup  will  have  to  follow  this  initial  step.  | 

i 

II.  Evaluation  of  the  goodness-of-fit  of  model  estimation. 

A.  Begins: 

The  planning  for  data  collection  and  reduction,  the  first  step 
of  this  task,  can  begin  itimediately  after  adoption  of  the  validation 
plan  by  the  Model  Validation  Group. 

B.  Methodology: 

1.  Planning  and  design  of  data  collection  and  reduction  activities 

as  described  in  final  validation  plan 

a.  Specific  work  assignments  to  participants 

b.  Schedule  of  activities 

c.  Required  equipment  and  data  collection  forms 

d.  Specification  of  data  format  for 

(1)  inputs  for  model  execution 

(2)  outputs  for  comparison  in  the  model  estimates 

e.  Number  of  days  on  which  to  collect  data 

f.  Sample  size  on  each  day 

g.  Measurement  (observation)  techniques  to  be  employed 

h.  Procedure  for  assembling  data  for  reduction 


L 
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i.  Data  reduction  procedures 

j.  Procedure  for  preparing  data  for  comparison  with  model  estimates 

2.  Execution  of  data  collection  and  reduction 

3.  Execution  of  model  runs  with  input  data 

4.  Comparison  of  collected  and  reduced  data  with  estimates  produced  by 
the  model  runs 

a.  Statistical  hypothesis  tests 

b.  Subjective  evaluation  of  tabular  and  graphic  comparisons 

5.  Decision  as  to  model's  acceptability  for  Phase  II 

C.  Preparations 

Final  selection  of  comparison  variables,  method  of  comparison 
and  comparison  criteria--Final  Model  Validation  Plan. 

D.  Simultaneous  Activities: 

The  planning  of  the  data  collection  and  reduction  can  begin 
simultaneously  with  the  contractor  disclosure  of  model  logic,  inputs, 
and  outputs.  However,  parts  of  the  data  collection  plan,  namely  the 
detailed  format  for  the  comparison  data  and  the  format  for  model  inputs 
[see  items  B-l-d-(l)  and  (2)]  , will  have  to  be  done  after  the 
group  has  been  exposed  to  the  details  of  the  model  logic,  assumptions, 
inputs  and  outputs. 

It  is  possible  to  do  the  sensitivity  analysis  evaluation  at  the 
same  time  as  the  goodness-of-fit  evaluation.  It  may,  however,  be  de- 
sirable to  do  the  goodness-of-fit  evaluation  after  this  step  because 
it  is  desirable  to  be  thoroughly  familiar  with  the  workings  and  output 
of  a model  before  performing  a sensitivity  analysis  of  it. 
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E.  Subsequent  Activities 

Based  on  the  above  discussion,  it  is  recommended  that  we  do  the 
sensitivity  evaluation  after  the  goodness-of-fit  evaluation. 

Ill,  Evaluation  of  Model  Fine-Grained  Sensitivity 

A.  Begins: 

Sensitivity  evaluation  should  begin  after  or  simultaneously  with 
the  evaluation  of  the  goodness-of-fit  of  the  model  (preceding  step). 

Both  of  these  steps  require  input  data  and  model  computer  runs. 

B.  Methodology 

The  working  subgroup  of  8 persons  should  specify  the  experiments 
to  be  performed  by  the  contractor.  Each  experiment  must  specify 
which  parameters  to  be  held  fixed,  which  to  vary  systematically, 
and  which  to  view  as  response  variables.  The  contractor  should  then 
execute  the  experiments  by  making  the  necessary  runs  of  the  computer 
model.  Some  additional  runs  may  be  made  by  the  Model  Validation  Group 
using  the  NAFEC  computer.  The  subgroup  then  evaluates  the  results 
of  the  experiments. 

C.  Preparations: 

The  working  subgroup  should  be  familiar  with  the  model  logic,  inputs 
and  outputs. 

D.  Simultaneous  Activities: 

The  final  goodness-of-fit  comparisons  and  evaluations  can  be  done 
at  the  same  time  as  this  step. 

E.  Subsequent  Activities: 

Final  evaluation  of  model  sensitivity 
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* 

A SIMULATION  MODEL  FOR  AIR  TRAFFIC  CONTROL  COMMUNICATIONS 
In  an  FAA  sponsored  study  of  Air  Traffic  Control  communications,  a purely 
analytical  model  of  the  ATC  system  appeared  unattainable.  As  a consequence, 
to  represent  the  actual  system  a simulation  model  was  developed  employing 
GPSS  V simulation  language  with  an  IBM  360/91  computer  facility.  The 
research  results  are  relevant  to  the  (i)  evaluation  of  the  efficiency  of  the 
present  communications  performance;  (ii)  measurement  of  the  capacity  of  the 
present  communications  channels;  and  (iii)  experimentation  with  various 
proposed  changes  in  the  control  structure. 

Structure  of  the  Simulation  Model 

ATC  communications  associated  with  101  control  sectors  in  the  New  York 
metroplex,  which  comprised  12  different  control  functions,  were  originally 
recorded  on  voice  tapes  for  a busy  afternoon  period  on  April  30,  1969,  and 
were  subsequently  sorted  and  digitalized  for  computer  analysis.  (Each 
control  sector  was  assigned  a radio  channel  of  a specified  frequency,  and 
the  conversations  between  the  controller  and  aircraft  pilots  were  open  to 
all  who  tuned  to  the  frequency.)  This  large  data  bank  was  analyzed  to  provide 
a statistical  basis  for  analysis  of  the  complex  communications  system.  The 
available  data  were  digested  in  many  ways,  and  whereever  possible  mathematical 
models  postulated  and  fitted  to  historical  events  that  served  as  components 
to  the  larger  system.  The  models  were,  of  course,  abstractions  of  various 
elements  in  the  historical  data,  and  as  each  was  derived  it  was  tested  to 
determine  its  adequacy  as  a replacement  for  the  data. 

*S0URCE;  Hsu,  D.  A.  and  Hunter,  J.  S.,  "Analysis  of  Simulation-Generated 
Responses  Using  Autoregressive  Models,"  paper  accepted  for  Management  Science,  1977, 
pp.  16-20. 
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In  this  study  the  major  responses  to  be  simulated  are: 

(i)  aircraft  loading,  n^,  number  of  aircraft  present  in  sector 
at  time  t; 

(ii)  channel  utilization,  C^,  proportion  channel  time  employed  at 
time  t; 

(iii)  number  of  aircraft  in  queue  waiting  to  communicate,  Q^. 

Each  of  these  responses  is  a time  series  reflecting  the  ebb  and  flow  of 
air  traffic  through  a sector  and  the  resulting  burden  of  communications. 


Validation  of  the  Simulator 

The  validation  of  the  simulation  model  depends  upon  the  two 
responses,  aircraft  loading,  n^,  and  channel  utilization,  C^.  Both  these 
responses  are  available  historically  and  both  are  generated  by  the 
simulation  model  as  time  series.  (The  series  cannot  be  obtained  from 
a real  system,  and  was  one  of  the  reasons  for  the  computer  simulations.) 
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The  essential  validation  step  consists  of  the  inference  that  the 
structure  of  the  observed  time  series,  and  the  structure  of  the  simulated 
time  series,  for  both  responses,  are  identical. 

To  supply  an  example,  the  observed  and  simulated  time  series  plots 
(both  recorded  for  a two-hour  period  and  averaged  over  each  nonoverlapping 
60-second  interval)  of  aircraft  loadings,  n^,  for  one  of  the  busiest  Low 
Altitude  Transitional  (LT)  sectors  in  the  N.  Y.  area.  Sector  453,  were 
compared  (see  Figs.  D-1  and  D-2).  As  the  first  step  in  characterizing  these 
two  series,  the  sample  autocorrelation  functions  of  both  were  obtained  (see 
Figs.  0-3  and  D-4).  An  inspection  of  the  two  estimated  autocorrelation  func- 
tions suggested  that  they  were  very  much  alike  and  both  exhibited  a damped 
sinusoid  pattern  peculiar  to  the  AR(2)  model.  The  AR(2)  model  was  thus 
fitted  and  the  two  autoregressive  parameters  estimated  for  both  series 
using  the  ordinary  least-squares  metnod.  Various  diagnostic  tests,  following 
the  procedure  outlined  in  Box  and  Jenkins  [2],  indicated  that  the  AR(2) 
model  fitted  both  sets  of  data  satisfactorily.  In  addition,  the  residual 
distributions  were  checked  and  found  to  be  consistent  with  the  normality 
assumption. 

As  a consequence  of  repeated  applications  of  these  inferential  procedures, 
conside''ab1e  confidence  has  been  generated  in  the  simulation  model.  There 
are  occasional  individual  sectors  for  which  validation  has  proved  impossible, 
a failure  generally  attributable  to  the  paucity  of  data  for  these  sectors, 
to  an  unusually  large  number  of  maverick  observations  which  make  the  distri- 
bution assumptions  untenable,  or,  to  the  pronounced  lack  of  independence 
of  the  arrivals  of  incoming  aircraft.  However,  the  vast  majority  of  the 
individual  sectors  (comprising  the  enroute,  the  local  control  the  local  and 
ground  control,  tne  radar  departure,  the  radar  arrival,  and  the  radar 
arrival -departure  control  functions)  have  been  successfully  simulated 


and  validated. 


□0*01 


(Source:  Report  No.  FM-RD-74--203) 
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Figure  o- 


Aufocorrelotion  Function  of  Historical 
Aircraft  Looding  Series  (Sector  453) 
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Figure  Autororreki  v'.n  Functior.  of  Simulated 

Aircraft  Loocing  Series  (Sector  453) 
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